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ABSTRACT 

The Research on Trial technique is designed to enable 
students to think critically about psychological research, to help 
them apply what they have learned in class in an in-depth way to this 
research, and to create a classroom environment in which research 
issues are debated. The technique employs a courtroom trial 
role-play, with students assigned to either defend or "prosecute" a 
published research article. In the trial, four students serve as 
defense lawyers for the article and another four as the prosecution, 
while the teacher acts as judge, ensuring fair proceedings* The 
teacher also provides the jury with guidelines on deciding the 
outcome of the trial, encouraging them to focus on the internal 
validity of the study's methodology and conclusions and the external 
validity or the study ! s applicability to other populations* A fall 
1994 evaluation of student participants' perceptions of the technique 
indicated that students were generally positive about the experience, 
although there were reports of some group members contributing more 
than others and of confusion on the part of the jury* Areas that need 
to be addressed with the technique include dealing with interpersonal 
tens ions that may arise between the role-playing students and 
providing clear instructions to the jury. Graphs of reponses from the 
student evaluation of the technique are appended* (BCY) 
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Research On Trial: 
A Pedagogy for Research Methods Instruction 

Goals of the Exercise 

The goal of Research on Trial (ROT) is threefold: A) to get students to think critically 
about psychological research, B) to apply what they have learned in class in an in-depth way to 
this research, and C) to create a classroom environment in which research issues are debated in an 
engaging way. 

This is done by using a courtroom trial situation to get students to closely examine the 
strengths and weaknesses of a published research article in psychology. In their assignment to 
either defend or "prosecute" an article, they must delve into the article deeply, looking for various 
problems in the design of the research, the possible threats to internal and external validity, 
measurement issues such as reliability and validity, statistical errors, etc. Simply asking one 
student to critique an article often proves unsatisfactory in terms of that students' motivation to 
truly examine the weaknesses of an article, because students often think that if a study is 
published, it is probably very good. However critiquing an article is similar to looking at it much 
as a lawyer must look at a case. The author of an article presents his/her case ("defends" it), and 
the critical reader must serve as prosecutor (looking for weaknesses). Finally, the reader is also 
the jury: left to think about the study and decide whether or not they think the study was done 
well, and the results can be relied upon. 

Thus, a debate situation, in which students arc asked to critique an article and where they 
would know that their statements would be challenged seems an ideal way to get students to 
critique an article in greater depth. Thus the Research On Trial simulation has some students 
defend an article, some students prosecute the article, and the rest of the class hear all the 
evidence and then deciding on whether they think the research was well conducted and the results 
arc believable (internally valid) or the study was not well conducted and the results cannot be 
believed. 

Student Instructions: 

The purpose of this role-play is to learn more about what makes good research by delving in 
detail into the "case" that the author(s) of an article try to make about the research they have 
conducted. The authors of any published research article try to convince the reader of the validity 
of their conclusions, i.e., the significant results they claim to have found. Consumers of research 
cannot read the results of these articles without keeping a critical eye ou how the research was 
conducted. Should you believe what the author claims? Should important decisions concerning 
people who are in need be based upon this research? As psychologists, you will be called upon to 
make such decisions in your work. Developing a knowledgeable, insightful critical eye is 
important for psychologists, let alone any educated person. 

3 
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This simulation is intended to teach and develop skills in being a critical thinker and consumer of 
research. Since it will be necessary to take liberties with the actual way a courtroom trial is run in 
order to adapt the scenario to our needs, the simulation will not attempt to teach students about 
the proper way a courtroom is run. 

The Proceedings 

Here's how we will conduct the trial: 

1 . Four students will represent the authors as defense lawyers , and four students will serve as 
lawyers for the prosecution . The other students in class will serve as a jury and hear the trial and 
decide whether they believe the defense that the author's conclusions are valid, or that they side 
with the prosecution that the author's conclusions are not valid. 

2. The judge (myself) will make sure the trial proceeds fairly. I will also instruct the jury as to 
what they are to decide once the proceedings are over. 

3. The Presentations of Defense and Prosecution Lawyers. The defense will present their case 
first to the court. Each person in the group should read carefully the entire article and the group 
should decide how to best present each section to the class. The defense will concentrate on the 
article's strengths, although it will serve them well to know the article's weaknesses as well. On 
the day of the presentation I will randomly assign each member of the group to present a section 
of the article (Introduction, Method, Results, or Discussion) to the class. Each student will 
present to the class a summary of that section of the article. You MUST use some form of media 
to help make your case as clearly and convincingly as possible to the jury (e.g., posters, drawings, 
handouts, overhead transparencies, etc.). The rest of the defense lawyers will continue by 
summarizing the other sections of the article. Posters or drawings on the blackboard are 
particularly encouraged for the Method and Results sections so that the everyone understands 
what tests were given when, to what group, and what the results indicate. When the defense 
lawyers have completed presenting their case (plan on 15 minutes maximum), I will randomly 
assign each of the prosecution lawyers to come up and present their side of the story, discussing 
each section separately and presenting what they believe are very real concerns about the study. 

4. The Summation . After the lawyers for the prosecution section have presented their 
observations, the defense will have the final word. They can address the criticisms made by the 
prosecution and reiterate what they feel are the strengths of the article. 

5. Jury Deliberations . After the defense summation the judge will explain to the class exactly what 
decisions they will have to return a verdict on. The jury will be allowed to discuss for 
approximately 1 0 minutes and then will be asked for their decisions. Unanimity of decision is not 
necessary. Please record the vote at the time that I call for the decision. We will then discuss the 
verdicts. 




The Roles 



Attorneys for the Defense 

The job of the "defense attorneys" is to advocate for the authors) and try to make the case to the 
jury that the authors' claims about the results of their research are valid ones. 

Everyone in the group should read the article and be familiar with each section so that: 

1 . you can help each other to build a convincing case. Try to win the case by presenting 
the research in its best light. This also means paying attention to the weaknesses as \* ell as the 
strengths of the article. What argument will you use to defend the author against the attacks of 
the prosecution if you (and they) see a weakness in the research? You may break up the article 
into four parts (Introduction, Methods, Results, and Discussion) but everyone in the group should 
be prepared to discuss any section because on the day of the trial I will randomly assign each 
member of the group to the section I want them to present to the class. 

2. you can defend it from the questions/criticisms that will be made against it by the 
prosecution. Know the strengths and weaknesses of the article. 

Attorneys for the Prosecution 

The job of the "prosecution attorneys" is to examine the research carefully and pay particular 
attention to the weaknesses and possible confounds in every section of the study. 

Everyone in the group should read the article and be familiar with each section so that: 

1 . you can help each other to build a convincing case. Try to win the case by poking holes 
in the research. This means paying particular attention to the weaknesses of the article. What 
arguments will you use to place doubt in the-jury's mind about the validity of the author's 
conclusions? You may break up the article into four parts (Introduction, Methods, Results, and 
Discussion) but everyone in the group should be prepared to discuss any section because on the 
day of the trial I will randomly assign each member of the group to the section I want them to 
present to the class. 

2. you can expertly pinpoint possible areas of weakness in the research. Prepare your 
critiques, and be ready to try to convince the jury that the problems you have found arc serious 
enough to place in doubt the conclusions of the authors. 

General Note: Think about the article and come up with your own questions about the research. 
What do you see as strengths and weaknesses of their approach? It is very helpful to try 
imagining yourself as a subject in this experiment. What would you experience? 




The Jury 



The class should listen carefully to the arguments presented by both sides. Carefully study the 
evidence that is given to you. Your job is to decide whether or not the author has done a good 
piece of research. No study is ideal , but can you have confidence in the results of this research? 
Feel free to raise your hand at any point in the trial. 



Grading procedure: Your grade is NOT based upon whether the jury agrees with the defense or 
the prosecution. I will pay attention to your presentation looking mostly for your ability to 
explain the article CLEARLY . This will not be easy. Remember that the jury will not have read 
the article. Recall that some form of assistance to the jury, in the form of drawings, posters, 
transparencies, or handouts which clarify the predictions, the research design, the results, etc., are 
REQUIRED. After the presentation I will ask each member of the group to give each other 
member of their group, and themselves, a letter grade based upon their contribution to the 
presentation. I will consider all this information in assigning each student a grade for this project. 

To assist you in critiquing the articles, keep in mind the following threats described by Campbell 
and Stanley (1963): 

Internal Validity 

At issue: How confident are you that the author's conclusions really do represent the effect of the 
I.V. on the D.V? i.e., are the author's conclusions solid, believable, convincing ones? 

1. History: Did anything unusual happen during the course of the experiment that 

could have affected the D.V.? 

Look for: a long time between the pre-test and post. 

2. Maturation: Are the results possibly due to natural changes in the subjects over time? 

For example, are subjects much elder at the end of the study? More tired? 

Look for: a long time between the pre-test and post 

3. Testing: If there was a pre-test, could it have affected the post-test in any way? 

Look for: Subjects who are unfamiliar with taking tests, who may have gotten better at taking 
them a second time, the same test used both at the pre-test and the post-test. 
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4. Instrumentation: Examine all the measurement instruments used in the article. Are 

they good instruments (reliable and valid)? 

Look for: Reliability and validity information? (i.e., alpha coefficients, test/retest info, etc.). Were 
they "home grown" instruments? Were people used to measure things? (Inter-judge reliability) 

5. Regression: If subjects were Jiosen to participate in the study on the basis of extreme 

scores, could scores have naturally moved toward the mean upon a re-test? 

Look for: Subjects chosen because they were poor/very good readers, had low self-esteem, etc. 

6. Selection: How were subjects were selected? Was it random? If not, what problems 

could occur as a result of the selection strategy used? 

Look for: Information regarding how the subjects were chosen to be in the study. 

7. Mortality: Did any subjects drop out? Could this have affected the results? 

Look for: How many subs took the pre-tcst and how many took the post-test? Are there unequal 
n sizes at the data collection points? Small n sizes may violate assumptions of normality of t and f 
tests. Also: statistical significance using a large N (ex: 1,000 subs) may not be practically 
significant. 



External Validity 



Q: If we assume that the author's conclusions are valid, the next question is: will the findings 
apply to anyone else? Are the findings unique to just the subjects used in the study, or will they 
generalize to other populations? 

Consider the sources of potential invalidity outlined by Stanley and Campbell. 



1. Interaction of testing and X: 



2. Interaction of selection and X: 



3. Reactive arrangements: 



ERLC 



Does it appear that the treatment works, but 
only if a pretest is given? If so, the treatment 
may not work elsewhere unless a pretest is 
also given. 

Does it appear that the treatment works, but 
because of the selection strategy used or type 
of subject, it may work only for these 
particular subjects? 

The treatment works, but only if the subject 
is in an experimental situation and the 
treatment may not work without all the 
"trappings" of being a research project. 

7 



Effectiveness of ROT 



As yet, no experimental study has been carried out on the technique. However, the 
technique did receive an evaluation in the Fall of 1994. Students who had just participated in it 
were asked to fill out questionnaires regarding their assessment of the efficacy of the approach. 
These results are shown in Appendix A. Student reaction was quite favorable. Some were 
apprehensive about giving an oral presentation to class, but afterward agreed that it was a 
valuable experience. They do report typical problems with a group project: some members 
contributing more than others, for example. It is hoped that students will feel it important to learn 
each section of the article and contribute to the group process since they know that the teacher 
will only assign which student is to present which section of the article on the day of the 
presentation. Students will need to exercise and develop their group process skills in order to 
make the project work well. Jury members found the experience confusing at times. This is 
discussed in the section below. Each semester that the ROT has been assigned, student feedback 
has used to refine the technique. 

Areas That Need Attention 

The ROT tends to involve students to the point where the prosecution may become overly 
critical, or the defense may become overly defensive or sensitive to critical comments. When 
students assume roles, they can take these roles very seriously, as the now famous Stanford prison 
study showed (Haney & Zimbardo, 1977). Thus, interpersonal tension has been noted between 
defense and prosecution teams. This has to be addressed both before, during, and after the ROT 
so that such tensions do not detract from the goal of the project. 

The jury may find it difficult to follow the description of the research because students are 
sometimes not skilled at making oral presentations. It is important that overheads or handouts be 
required from each set of lawyers ("exhibits A and B") so that the jury is assisted in their 
understanding of what has occurred in the article. 

Conclusion 

The Research On Trial technique appears to be an effective way to involve students in examining 
published research in an in-depth manner, and in getting them to apply what they have learned to a 
specific piece of research. They learn that no study is perfect, and that compromises have to be 
made along the way. They also learn in a manner that results in stimulating class discussion and 
debate. 
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Appendix A 
Student Reaction to Research On Trial 



Did you understand the ROT instructions? 



Not Too Wei! 




Very Wei! 
83% 



How helpful was the oral defense/prosecution in ROT? 




10 

-132- 



How effective was the group process In ROT? 



Somewhat 

Effective Not Effective 




5% 



Very Effective 
66% 



How helpful was being a Jury member In ROT? 




ii 
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